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Abstract 

This study developed a game-based formative assessment, called tic-tac-toe quiz for single-player version 
(TRIS-Q-SP), in an energy education e-learning system. This assessment game combined tic-tac-toe with online 
assessment, and revised the rule of tic-tac-toe for stimulating students to use online formative assessment actively. 
Additionally, to investigate which types of formative feedback were appropriate for use in this game-based formative 
assessment, immediate elaborated feedback (IEF) and no immediate elaborated feedback (NIEF) mechanisms were 
designed and incorporated in TRIS-Q-SP. The findings revealed that all participants improved their energy 
knowledge through the e-leaming system, and expressed enjoyment and positive opinions toward the TRIS-Q-SP. 
However, the various statistical results suggested that providing IEF for each question answered in TRIS-Q-SP was 
the optimal design. This design facilitated the enhancement of energy knowledge acquisition when comparing it with 
NIEF condition, and did not affect all participants’ enjoyment perceptions. 
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1. Introduction 

With the development of technology, instruction and learning through computers and the Internet has become an new 
educational trend in the world (Bentley, Selassie, & Shegunshi, 2012); more than 10 terms (Brown, Charlier, & 
Pierotti, 2012), such as distance learning, online learning, blended learning, mobile learning, and e-learning, which is 
to refer to instruction and learning delivered through computer technology, have evolved from this trend. In other 
words, educational courses are no longer confined to only traditional classrooms because online courses are also 
offered by schools and enterprises in contemporary educational systems. For example, more than 31 percent of U.S. 
college students took at least one online course in 2010 (Allen & Seaman, 2011). 

Online assessment is an important component in current e-learning systems (McKimm, Jollie, Cantillon, 2003), 
particularly, online formative assessment tools play a crucial role in e-learning environments. The tool can 
automatically present questions and scores and quickly provide feedback, immediately informing students of their 
learning performance (Liu, Wang, & Zhang, 1999). The tool also provides opportunities for enhancing system 
interactions when learners engage in individualized learning (McKimm, Jollie, & Cantillon, 2003; Vasilyeva, 
Pechenizkiy, & De Bra, 2008). Previous studies have suggested that providing online formative assessments 
undoubtedly improves e-leaming motivation and effectiveness (Gardner, Sheridan, & White, 2002; Henly, 2003; 
Khan, Davies, & Gupta, 2001). However, a related study indicated that not every online learner maintains their high 
motivation for using online formative assessment in online courses (Buchanan, 2000; Henly, 2003). In other words, 
although online formative assessment allows opportunities to repeat tests or practice at convenient times, certain 
learners could continue testing after attempting once or twice, specifically when learners were constantly presented 
with boring multiple choice tests. Thus, enhancing participation in online formative assessments has become a 
critical research topic (Costal, Mullan, Kothe, Butow, 2010; Lin & Lai, 2013). 

To address these problems, online formative assessments could combine games with assessments to enhance the 
participation ratio for using online formative assessments when numerous scholars (Vogel et al., 2006; Yang, Chien, 
& Liu, 2012) have claimed that game-based learning can enhance learning motivation and effectiveness. Although 
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some scholars (Tsai, 2013; Wang, 2008) have employed game-based formative assessments in e-leaming, research 
regarding this topic is scarce. Tsai (2013) combined multiple choice tests with multi-player tic-tac-toe game to be an 
online formative assessment (called as TRIS-Q) in e-leaming environment, and indicated that students’ learning 
effectiveness was enhanced after using online learning and game-based assessment. However, this study also found 
some drawbacks arised from a multi-player formative assessment, and suggested a single-player tic-tac-toe 
assessment game should be developed. 

Based on Tsai’s (2013) suggestion, this study attempted to develop a single-player formative assessment game, called 
tic-tac-toe quiz for single-player version (TRIS-Q-SP), in an e-learning system targeting knowledge of energy 
education. Basically, TRIS-Q-SP adopted the TRIS-Q’s game mles proposed by Tsai. However, in TRIS-Q-SP, a 
player only can play tic-tac-toe game against the computer, also known as the Artificial Intelligence (AI), with two 
levels of difficulty (easy and hard). Additionally, to provide effective feedbacks in formative assessment, TRIS-Q-SP 
adopted diverse types of formative feedback by referring to relevant studies (Shute, 2008) of formative feedback. 
The primary feedback comprised immediate feedback, delayed feedback, knowledge of results (KR), which inform 
learners of whether their answer is correct or incorrect, and elaborated feedback (EF), which present a detailed 
message, providing information and clues relevant to the question to guide learners toward the correct answers. 

Although various formative assessment studies (Van der Kleij, Timmers, & Eggen, 2011) have suggested that EF 
improves learning effectiveness, the immediate elaborated messages in TRIS-Q-SP may annoy game participants. 
Students could focus on completing game without reading the immediate elaborated messages in-game. Therefore, 
assuming that whether immediate EF negatively affects game-based assessments is unknown, this study adopted two 
types of immediate EF in TRIS-Q-SP, namely, immediate EF (IEF) and no immediate EF (NIEF), and subsequently 
explored how distinct feedback types affected the effectiveness of and sense of involvement in TRIS-Q-SP. 
Regarding e-leaming effectiveness, the study primarily investigated how various feedback types of game-based 
formative assessment influenced the acquisition of energy knowledge. Concerning sense of involvement, the 
researchers primarily explored learner enjoyment after using the game-based formative assessment tool. 


2. The E-Learning Environment and TRIS-Q-SP 


This study involved Flash, ActionScript, and database technologies, which were used to develop an e-leaming 
system and TRIS-Q-SP on energy education appropriate for junior high school students. The course material was 
designed based on the textbooks used in Taiwan. The contents were divided into three major themes: sources of 
energy, application of energy, saving energy and development of new energy. Figure 1 shows the e-leaming menu for 
sources of energy, which lists relevant materials for students to learn, explaining various sources of energy. After the 
learner selected the material, human-like voice or relevant animations appeared to captivate the learning interest of 
the students. 



Figure 1: The E-Leaming for Energy Education 

TRIS-Q-SP was designed in this e-leaming system to allow learners to participate whenever they desire. Learners 
could select the button for entering the online assessment in e-leaming system (as shown in Figure 1) and 
conveniently enter the main menu of TRIS-Q-SP (Figure 2). When learners enter the TRIS-Q-SP, they could click the 
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different buttons (as shown in Figure 2) for entering a low level game, starting a high level game, querying the high 
score list, querying the personal score, or querying answering history. 
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Figure 2: The Game Menu of TRIS-Q-SP 

As soon as learner evokes a new game, the game-based formative assessment starts shown as step 1 of Figure 3. The 
game rule is similar to the traditional tic-tac-toe game, namely, the first player to place three tokens in a row, column, 
or diagonal is the winner. However, this game changes some rules of traditional tic-tac-toe. That is, when a player 
takes his turn, the game will randomly pop a multiple-choice quiz which was selected randomly from database 
according to the e-learning contents, shown as step 2 of Figure 3. Since the victory always goes to the one who 
makes the first move in tic-tac-toe, in order to increase the difficulty and playfulness, the game rule is slightly 
modified. If the player responses the right answers, he can place his token (step 3 of Figure 3); otherwise he will 
place his opponent’s token (step 3 of Figure 3). Thus, the new game rule could stimulate participants to seriously 
answer the questions in TRIS-Q-SP. 
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Figure 3: The Process of Playing TRIS-Q-SP and Immediate Feedback 

Additionally, the TRIS-Q-SP integrated diverse types of formative feedback, such as immediate KR (IKR), 
immediate EF (IEF), delayed KR (DKR), and delayed EF (DEF). The step 3 of Figure 3 shows the screen that 
appeared when students finished answering a question and made a move in the TRIS-Q-SP. An IKR feedback 
showed on the right side of the screen, informing the students if they answered the previous question correctly. The 
previous question and IEF feedback also showed on the left side of the screen, offering relevant data and clues to the 
previous question. However, during the game, participants concentrating on the TRIS-Q-SP may neglect the IEF 
messages. To prevent the IEF messages from disturbing the students’ flow, one group did not receive immediate EF 
during the experiment. Figure 4 shows the screen that appeared when students entered the function of querying 
answering history in TRIS-Q-SP. The screen clearly displayed all the answered questions and detailed records 
regarding whether answers were correct (DKR). A DEF message also showed on the top side of the screen, offering 
relevant data and clues to the answered question. 



3. Method 

3.1 Participants and Procedures 

This research used the quasi-experiment study method. Fifty-five ninth-grade students selected from two classes of a 
junior high school in Taiwan, participated in this experiment; each class included 27 and 28 students respectively. In 
the experiment, all participants were randomly assigned by class to two groups: ‘immediate elaborated feedback’ 
group (IEF group, N=27, 14 males and 13 females) and ‘no immediate elaborated feedback’ group (NIEF group, 
N=28, 14 males and 14 females). 

The study duration was eight class periods (45 min each) that lasted for 4 weeks. Prior to participating in the 
e-learning activity, all participants were asked to complete energy knowledge pretest. After a brief introduction about 
how to log in to the e-leaming system and how to play the TRIS-Q-SP, students were instructed to engage with one 
learning topic per week according to the three major learning themes in the e-learning system. In last week of the 
experiment, the students were asked to comprehensively revise for the all learning themes. To prevent the students 
from engaging in the TRIS-Q-SP without reading the e-leaming contents, the suggestions of Tsai, Kinzer, Hung, 
Chen, and Hsu (2013) were followed. When beginning a new learning topic, the teachers asked the students to read 
the e-leaming materials for at least 30 minutes before engaging in the TRIS-Q-SP based on their grouping conditions. 
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After completing the e-leaming activity, all participants were asked to complete energy knowledge posttest and a 
questionnaire regarding their enjoyment perception of playing TRIS-Q-SP. 

3.2 Instruments 

3.2.1 Energy Knowledge Test 

To understand how student knowledge was affected after completing the e-leaming activity, energy knowledge test 
was developed based on the content of the e-leaming material. The test comprised 40 multiple-choice questions and 
the total score was 100. Two junior high school teachers vetted all items’ content validity, and the reliability was 
assessed by internal consistency (Kuder-Richardson 20, KR-20) through a pilot study with 58 ninth-grade students 
before the experiment. A KR-20 reliability coefficient of .85 was obtained. 

3.2.2 Questionnaire of Enjoyment Perception 

This study developed a questionnaire to measure students’ enjoyment perception regarding the two feedback 
conditions of TRIS-Q-SP. A 5-point Likert-type scale consisting of 4 items was developed by referring to the 
enjoyment scale of Down and Sunder (2011). To understand the authentic perceptions of the participants after using 
the game-based formative assessment, an open-ended item was also included for this questionnaire. It allowed the 
students to record their opinions and suggestions toward the TRIS-Q-SP. The Cronbach’s a of the inventory was .83. 


4. Results 

4.1 Effects on Knowledge Acquisition 

To examine how the e-learning course affected student knowledge acquisition, the energy knowledge pretest and 
posttest scores were employed in a paired sample Mest. Based on analysis of paired sample £-test, the energy 
knowledge pretest and posttest scores significantly differed for the NIEF group ( t (27) = 6.722, p = .000), indicating 
that the posttest scores (M = 75.54, SD =14. 15) were significantly higher than the pretest scores (M = 63.68, SD 
=15.49). The energy knowledge pretest and posttest scores also exhibited significant differences in the IEF group (t 
(26) = 14. 862, p = .000), indicating that the posttest scores (M = 74.17, SD = 13.27) were significantly higher 
compared with the pretest scores (M = 56.20, SD = 12). Based on these statistical data, all groups substantially 
improved their average energy knowledge assessment scores, regardless of which formative feedback they received. 
Figure 5 displays the score improvements for each group of students. 
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Figure 5: A Comparison of the Energy Knowledge Test Scores for the Two Groups 

To compare the learning effects between two formative feedback conditions in a game-based formative assessment, a 
one way analysis of covariance (ANCOVA) was employed to examine the differences between energy knowledge 
posttest scores of the two groups with subjects’ pretest scores as covariate. The test of homogeneity of regression 
showed that the homogeneity of regression of the two groups was not different (F (1,51) = .312, p = .579), indicating 
that the coefficient is homogeneous and fits the basic assumption of the ANCOVA. The results of ANCOVA 
indicated that after the influence of covariate was controlled, the post-test scores of the two groups were significantly 
different (F (1,52) = 4.594, p = .037, rj 2 =. 081). Furthermore, the post-hoc test indicated that after adjusting the 
mean, the average score of the energy knowledge posttest for the IEF group was 77.09, which was significantly 
higher compared with the 72.72 attained by NIEF group. In other words, compared with not providing IEF messages, 
providing IEF messages during game-based assessment was more beneficial for improving the effectiveness of 
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student knowledge acquisition during e-learning. 

4.2 Evaluation of Enjoyment Perception 

To evaluate learner’s enjoyment perception toward the game-based assessment, descriptive statistics were used to 
analyze their scores on the enjoyment perception questionnaire, which used a 5-point Likert-type scale ranging from 
1 (strongly disagree) to 5 (strongly agree). Table 1 shows the scores of the enjoyment perception questionnaire for 
the two groups of students. The average score for the four questions was 3.62; each question achieved a score 
exceeding 3, indicating positive student views regarding enjoyment of TRIS-Q-SP. 


Table 1: The Mean Scores for the Enjoyment Questionnaire 


Questions 

NIEF 

IEF 

Total 

I enjoyed playing the TRIS-Q-SP 

3.93 

3.79 

3. 86 

I enjoyed the competition while playing the TRIS-Q-SP 

3.75 

3. 61 

3.68 

I think this was an exciting game experience 

3.71 

3.55 

3. 63 

I would be happy to play similar games in the future when opportunities allow 

3. 18 

3.41 

3.30 

Total 

3. 64 

3. 51 

3. 62 


To explore whether different feedback types affected student enjoyment toward the game-based assessment, this 
study conducted the independent /-test by using feedback types as the independent variables and the enjoyment score 
as the dependent variable. The analysis of the independent /-test indicated that while the average enjoyment 
perception score (M = 3.64, SD =1.00) on NIEF group was higher than the one (M = 3.51, SD =. 81) on IEF group, 
there was no significant difference between the two groups (/=.541 , p=. 591). The result suggests that providing 
immediate elaborated feedback during gameplay may not affect students’ enjoyment perception when comparing it to 
the NIEF group. 


5. Discussion 

Based on Tsai’s (2013) research, this study developed a tic-tac-toe-based formative assessment for single-player 
mode in an energy education e-leaming system. Additionally, to investigate which types of feedback messages were 
appropriate for use in this game-based formative assessment and how distinct types of feedback messages affected 
e-leaming effectiveness and learner perceptions, IEF and NIEF feedback mechanisms were designed and 
incorporated in TRIS-Q-SP game based on theories relevant to feedback types suitable for formative assessments. 

The findings revealed that all participants improved their energy knowledge through the e-learning activity. The 
results of the dependent sample /-test indicated that regardless of the feedback type of TRIS-Q-SP game, the posttest 
scores for the energy knowledge test were higher than the pretest scores. Thus, the proposed online energy 
educational course that incorporated game-based formative assessment was effective in improving the energy 
knowledge for junior high school students. The result was consistent with previous researchers’ findings (Wang, 
2008; Tsai, 2013) indicating that game-based formative assessment can improve learning effectiveness. 

This study also found that if the system provided immediate elaborated feedback during the TRIS-Q-SP game, it 
significantly increased the level of learner performance in energy knowledge. Based on one-way ANCOVA results, 
the participants who received immediate detailed feedback scored higher on the energy knowledge posttest than did 
those who received no IEF. The findings of the study were consistent with previous studies that suggested immediate 
feedback and elaborated feedback enhanced learning (Corbett & Anderson, 1989; Dihoff, Brosvic, Epstein, & Cook, 
2003; Meyer et al., 2010; Van et al., 2011). 

Furthermore, based on the average scores on the enjoyment scale, most students expressed positive perceptions 
toward the TRIS-Q-SP game, indicating that the game was popular among the students. Also, there was no 
significant difference between the two groups. The various opinions and comments that the students provided to 
open-ended questions suggested that most students thought the game was fun, leaving comments such as “I think the 
game is fun and exciting, but I wish there was more time,” “I think this game is great! The game was different from 
what I imagined. To decide putting an O or X based on answering the questions correctly was very exciting,” 
“Although I couldn’t achieve high score list, I hope I can play such game in the future.” Meanwhile, students’ 
gameplay behaviors in TRIS-Q-SP also supports that this game was fun and enhanced the participation ratio for 
using online formative assessments. According to the game usage derived from system logs, in average the IEF and 
NIEF group played 210 and 182 tournaments respectively on TRIS-Q-SP during the experiment. The result was 
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consistent with previous researchers’ findings (Wang, 2008; Tsai, 2013) indicating that game-based formative 
assessment can enhance learning motivation. 


6. Conclusion and future work 

In summary, the proposed TRIS-Q-SP, regardless of feedback type, facilitated e-learning effectiveness. The students 
expressed enjoyment and positive opinions. However, the various statistical results suggest that providing IEF for 
each question answered in the TRIS-Q-SP game was the optimal design. This design facilitated the enhancement of 
energy knowledge acquisition. Additionally, these feedback messages did not affect participant enjoyment. In the 
future, the TRIS-Q-SP can be compared with the TRIS-Q proposed by Tsai (2013) to investigate the learning 
effectiveness between single-player and multi-player assessment game. Also, future studies can employ the 
TRIS-Q-SP game in various e-leaming courses to verify the effectiveness of the current study. 
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